Optical systems with lens-based static foveation

ABSTRACT

An electronic device may include a display module that produces light having an image, a lens that directs the light to a waveguide, and a waveguide that directs the light to an eye box. The lens may produce a foveated image in the light by applying a non-uniform magnification to the image in the light. The non-uniform magnification may vary as a function of angle within a field of view of the lens. This may allow the foveated image to have higher resolution within the central region than in the peripheral region. Performing foveation using the lens maximizes the resolution of images at the eye box without increasing the size of the display module. Control circuitry on the device may apply a pre-distortion to the image that is an inverse of distortion introduced by the lens in producing the foveated image.

This application is a continuation of international patent applicationNo. PCT/US2020/050566, filed Sep. 11, 2020, which claims the benefit ofU.S. provisional patent application No. 62/901,412, filed Sep. 17, 2019,which are hereby incorporated by reference herein in their entireties.

BACKGROUND

This relates generally to optical systems and, more particularly, tooptical systems for displays.

Electronic devices may include displays that present images close to auser's eyes. For example, devices such as virtual reality and augmentedreality headsets may include displays with optical elements that allowusers to view the displays.

It can be challenging to design devices such as these. If care is nottaken, the components used in displaying content may be unsightly andbulky and may not exhibit desired levels of optical performance.

SUMMARY

An electronic device such as a head-mounted device may have one or morenear-eye displays that produce images for a user. The head-mounteddevice may be a pair of virtual reality glasses or may be an augmentedreality headset that allows a viewer to view both computer-generatedimages and real-world objects in the viewer's surrounding environment.

The near-eye display may include a display module that generates lightand an optical system that redirects the light from the display moduletowards an eye box. The optical system may include a waveguide having aninput coupler and an output coupler. The optical system may include alens that directs the light from the display module towards thewaveguide. The display module may include a reflective display panel, anemissive display panel, or other display hardware.

The lens may perform static foveation operations on the light producedby the display module. For example, the light generated by the displaymodule may include an image. The lens may produce a foveated image byapplying a non-uniform magnification to the image in the light. Thenon-uniform magnification may vary as a function of angle within a fieldof view of the lens. For example, the lens may apply more magnificationto a peripheral region of the field of view, and thus the image, than toa central region of the field of view. This may allow the foveated imageto have a higher resolution within the central region than in theperipheral region. Performing foveation using the lens maximizes theresolution of images at the eye box without increasing the size of thedisplay module. Control circuitry on the device may apply apre-distortion to the image prior to the image being displayed by thedisplay module. The pre-distortion may be an inverse of distortionintroduced by the lens in producing the foveated image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an illustrative system having a display inaccordance with some embodiments.

FIG. 2 is a top view of an illustrative optical system for a displayhaving a lens that performs static foveation operations on image lightprovided to a waveguide in accordance with some embodiments.

FIG. 3 is a top view of an illustrative reflective display that may beused to provide light to a lens of the type shown in FIG. 2 inaccordance with some embodiments.

FIG. 4 is a diagram of an illustrative statically foveated image thatmay be output by a lens of the type shown in FIG. 2 in accordance withsome embodiments.

FIG. 5 is a plot of pixel density (pixels-per-degree) as a function offield of view angle for a statically foveated image that may be outputby a lens of the type shown in FIG. 2 in accordance with someembodiments.

FIG. 6 is a plot of magnification as a function of field of view anglefor a lens of the type shown in FIG. 2 in accordance with someembodiments.

FIG. 7 is a diagram of an illustrative lens that performs staticfoveation operations on image light in accordance with some embodiments.

FIG. 8 is a flow diagram showing how illustrative control circuitry mayperform pre-distortion operations on an image to mitigate subsequentdistortion by a lens that performs static foveation operations on theimage in accordance with some embodiments.

FIG. 9 is a plot showing how light-emitting elements may beindependently controlled as a function of position to compensate foroff-axis intensity variations in an optical system of the type shown inFIGS. 1-3, 7, and 8 in accordance with some embodiments.

FIG. 10 is a flow chart of illustrative steps that may be performed by adisplay in providing statically-foveated images to an eye box inaccordance with some embodiments.

DETAILED DESCRIPTION

An illustrative system having a device with one or more near-eye displaysystems is shown in FIG. 1 . System 10 may be a head-mounted devicehaving one or more displays such as near-eye displays 14 mounted withinsupport structure (housing) 20. Support structure 20 may have the shapeof a pair of eyeglasses (e.g., supporting frames), may form a housinghaving a helmet shape, or may have other configurations to help inmounting and securing the components of near-eye displays 14 on the heador near the eye of a user. Near-eye displays 14 may include one or moredisplay modules such as display modules 14A and one or more opticalsystems such as optical systems 14B. Display modules 14A may be mountedin a support structure such as support structure 20. Each display module14A may emit light 22 (image light) that is redirected towards a user'seyes at eye box 24 using an associated one of optical systems 14B.

The operation of system 10 may be controlled using control circuitry 16.Control circuitry 16 may include storage and processing circuitry forcontrolling the operation of system 10. Circuitry 16 may include storagesuch as hard disk drive storage, nonvolatile memory (e.g.,electrically-programmable-read-only memory configured to form a solidstate drive), volatile memory (e.g., static or dynamicrandom-access-memory), etc. Processing circuitry in control circuitry 16may be based on one or more microprocessors, microcontrollers, digitalsignal processors, baseband processors, power management units, audiochips, graphics processing units, application specific integratedcircuits, and other integrated circuits. Software code (instructions)may be stored on storage in circuitry 16 and run on processing circuitryin circuitry 16 to implement operations for system 10 (e.g., datagathering operations, operations involving the adjustment of componentsusing control signals, image rendering operations to produce imagecontent to be displayed for a user, etc.).

System 10 may include input-output circuitry such as input-outputdevices 12. Input-output devices 12 may be used to allow data to bereceived by system 10 from external equipment (e.g., a tetheredcomputer, a portable device such as a handheld device or laptopcomputer, or other electrical equipment) and to allow a user to providehead-mounted device 10 with user input. Input-output devices 12 may alsobe used to gather information on the environment in which system 10(e.g., head-mounted device 10) is operating. Output components indevices 12 may allow system 10 to provide a user with output and may beused to communicate with external electrical equipment. Input-outputdevices 12 may include sensors and other components 18 (e.g., imagesensors for gathering images of real-world object that are digitallymerged with virtual objects on a display in system 10, accelerometers,depth sensors, light sensors, haptic output devices, speakers,batteries, wireless communications circuits for communicating betweensystem 10 and external electronic equipment, etc.).

Display modules 14A may include reflective displays (e.g., liquidcrystal on silicon (LCOS) displays, digital-micromirror device (DMD)displays, or other spatial light modulators), emissive displays (e.g.,micro-light-emitting diode (uLED) displays, organic light-emitting diode(OLED) displays, laser-based displays, etc.), or displays of othertypes. Light sources in display modules 14A may include uLEDs, OLEDs,LEDs, lasers, combinations of these, or any other desired light-emittingcomponents.

Optical systems 14B may form lenses that allow a viewer (see, e.g., aviewer's eyes at eye box 24) to view images on display(s) 14. There maybe two optical systems 14B (e.g., for forming left and right lenses)associated with respective left and right eyes of the user. A singledisplay 14 may produce images for both eyes or a pair of displays 14 maybe used to display images. In configurations with multiple displays(e.g., left and right eye displays), the focal length and positions ofthe lenses formed by components in optical system 14B may be selected sothat any gap present between the displays will not be visible to a user(e.g., so that the images of the left and right displays overlap ormerge seamlessly).

If desired, optical system 14B may contain components (e.g., an opticalcombiner, etc.) to allow real-world image light from real-world imagesor objects 25 to be combined optically with virtual (computer-generated)images such as virtual images in image light 22. In this type of system,which is sometimes referred to as an augmented reality system, a user ofsystem 10 may view both real-world content and computer-generatedcontent that is overlaid on top of the real-world content. Camera-basedaugmented reality systems may also be used in device 10 (e.g., in anarrangement which a camera captures real-world images of object 25 andthis content is digitally merged with virtual content at optical system14B).

System 10 may, if desired, include wireless circuitry and/or othercircuitry to support communications with a computer or other externalequipment (e.g., a computer that supplies display 14 with imagecontent). During operation, control circuitry 16 may supply imagecontent to display 14. The content may be remotely received (e.g., froma computer or other content source coupled to system 10) and/or may begenerated by control circuitry 16 (e.g., text, other computer-generatedcontent, etc.). The content that is supplied to display 14 by controlcircuitry 16 may be viewed by a viewer at eye box 24.

FIG. 2 is a top view of an illustrative display 14 that may be used insystem 10 of FIG. 1 . As shown in FIG. 2 , near-eye display 14 mayinclude one or more display modules such as display module 14A and anoptical system such as optical system 14B. Optical system 14B mayinclude optical elements such as one or more waveguides 26. Waveguide 26may include one or more stacked substrates (e.g., stacked planar and/orcurved layers sometimes referred to herein as waveguide substrates) ofoptically transparent material such as plastic, polymer, glass, etc.

If desired, waveguide 26 may also include one or more layers ofholographic recording media (sometimes referred to herein as holographicmedia, grating media, or diffraction grating media) on which one or morediffractive gratings are recorded (e.g., holographic phase gratings,sometimes referred to herein as holograms). A holographic recording maybe stored as an optical interference pattern (e.g., alternating regionsof different indices of refraction) within a photosensitive opticalmaterial such as the holographic media. The optical interference patternmay create a holographic phase grating that, when illuminated with agiven light source, diffracts light to create a three-dimensionalreconstruction of the holographic recording. The holographic phasegrating may be a non-switchable diffractive grating that is encoded witha permanent interference pattern or may be a switchable diffractivegrating in which the diffracted light can be modulated by controlling anelectric field applied to the holographic recording medium. Multipleholographic phase gratings (holograms) may be recorded within (e.g.,superimposed within) the same volume of holographic medium if desired.The holographic phase gratings may be, for example, volume holograms orthin-film holograms in the grating medium. The grating media may includephotopolymers, gelatin such as dichromated gelatin, silver halides,holographic polymer dispersed liquid crystal, or other suitableholographic media.

Diffractive gratings on waveguide 26 may include holographic phasegratings such as volume holograms or thin-film holograms, meta-gratings,or any other desired diffractive grating structures. The diffractivegratings on waveguide 26 may also include surface relief gratings formedon one or more surfaces of the substrates in waveguides 26, gratingsformed from patterns of metal structures, etc. The diffractive gratingsmay, for example, include multiple multiplexed gratings (e.g.,holograms) that at least partially overlap within the same volume ofgrating medium (e.g., for diffracting different colors of light and/orlight from a range of different input angles at one or morecorresponding output angles).

Optical system 14B may include collimating optics such as collimatinglens 34. Lens 34 may include one or more lens elements that help directimage light 22 towards waveguide 26. If desired, display module 14A maybe mounted within support structure 20 of FIG. 1 while optical system14B may be mounted between portions of support structure 20 (e.g., toform a lens that aligns with eye box 24). Other mounting arrangementsmay be used, if desired.

As shown in FIG. 2 , display module 14A may generate light 22 associatedwith image content to be displayed to eye box 24. Light 22 may becollimated using a lens such as collimating lens 34. Optical system 14Bmay be used to present light 22 output from display module 14A to eyebox 24.

Optical system 14B may include one or more optical couplers such asinput coupler 28, cross-coupler 32, and output coupler 30. In theexample of FIG. 2 , input coupler 28, cross-coupler 32, and outputcoupler 30 are formed at or on waveguide 26. Input coupler 28,cross-coupler 32, and/or output coupler 30 may be completely embeddedwithin the substrate layers of waveguide 26, may be partially embeddedwithin the substrate layers of waveguide 26, may be mounted to waveguide26 (e.g., mounted to an exterior surface of waveguide 26), etc.

The example of FIG. 2 is merely illustrative. One or more of thesecouplers (e.g., cross-coupler 32) may be omitted. Optical system 14B mayinclude multiple waveguides that are laterally and/or vertically stackedwith respect to each other. Each waveguide may include one, two, all, ornone of couplers 28, 32, and 30. Waveguide 26 may be at least partiallycurved or bent if desired.

Waveguide 26 may guide light 22 down its length via total internalreflection. Input coupler 28 may be configured to couple light 22 fromdisplay module 14A (lens 34) into waveguide 26, whereas output coupler30 may be configured to couple light 22 from within waveguide 26 to theexterior of waveguide 26 and towards eye box 24. For example, displaymodule 14A may emit light 22 in direction +Y towards optical system 14B.When light 22 strikes input coupler 28, input coupler 28 may redirectlight 22 so that the light propagates within waveguide 26 via totalinternal reflection towards output coupler 30 (e.g., in direction X).When light 22 strikes output coupler 30, output coupler 30 may redirectlight 22 out of waveguide 26 towards eye box 24 (e.g., back along theY-axis). In scenarios where cross-coupler 32 is formed at waveguide 26,cross-coupler 32 may redirect light 22 in one or more directions as itpropagates down the length of waveguide 26, for example.

Input coupler 28, cross-coupler 32, and/or output coupler 30 may bebased on reflective and refractive optics or may be based on holographic(e.g., diffractive) optics. In arrangements where couplers 28, 30, and32 are formed from reflective and refractive optics, couplers 28, 30,and 32 may include one or more reflectors (e.g., an array ofmicromirrors, partial mirrors, or other reflectors). In arrangementswhere couplers 28, 30, and 32 are based on holographic optics, couplers28, 30, and 32 may include diffractive gratings (e.g., volume holograms,surface relief gratings, etc.).

FIG. 3 is a diagram of display module 14A in a scenario where displaymodule 14A is a reflective-type display. As shown in FIG. 3 , displaymodule 14A may include an illumination source such as light source 36.Light source 36 may have one or more light-emitting components(elements) 35 for producing output light. Light-emitting elements 35 maybe, for example, light-emitting diodes (e.g., red, green, and bluelight-emitting diodes, white light-emitting diodes, and/orlight-emitting diodes of other colors). Illumination may also beprovided using light sources such as lasers or lamps.

In the example of FIG. 3 , display module 14A is a reflective displaymodule such as a liquid-crystal-on-silicon (LCOS) display module, amicroelectromechanical systems (MEMs) display module (sometimes referredto as digital micromirror devices (DMDs)), or other display modules(e.g., spatial light modulators). An optical component such as prism 42may be interposed between light source 36 and display panel 38. Displaypanel 38 may be, for example, an LCOS display panel, a DMD panel (e.g.,a panel having an array of micromirrors), etc. Optical components suchas polarizers, beam splitters, lenses, and/or other components may beinterposed between light source 36 and prism 42, between prism 42 anddisplay panel 38, and/or between lens 34 and prism 42.

Display panel 38 may include pixel array 40 (e.g., an array ofmicromirrors where each micromirror corresponds to a given pixel in theimage in scenarios where display panel 38 is a DMD panel). Asillustrated by light ray 22′, prism 42 may be used to coupleillumination from light source 36 to display panel 38 and may be used tocouple reflected image light from pixel array 40 of display panel 38 tolens 34. Lens 34 may be used to provide image light from display module14A (e.g., as light 22) to waveguide 26 of FIG. 2 . Lens 34 may have arelatively wide field of view (e.g., at least 52°×52°, at least 52° by30°, etc.).

The example of FIG. 3 is merely illustrative and, in general, displaymodule 14A may be implemented as an emissive display module (e.g.,having a uLED panel, etc.) or other types of display modules (e.g.,display modules having light projectors, scanning mirrors, etc.).Display module 14A may include multiple light sources 36 located at thesame and/or different sides of prism 42. Each light source 36 and/oreach light-emitting element 35 may be independently controlled (e.g.,may be independently activated or deactivated, emit light withindependently-controlled intensities, etc.). Each light source 36 mayinclude light-emitting elements 35 that emit light of the samewavelength range (e.g., color) or may include different light-emittingelements 35 that emit light in two or more different wavelength ranges(e.g., colors). The light sources 35 in each light source 36 may bearranged in an M-by-N array or in any other desired pattern if desired.

It may be desirable to display high resolution images using display 14.However, in practice, the human eye may only be sensitive enough toappreciate the difference between higher resolution and lower resolutionimage data near the center of its field of view (e.g., a user may beless sensitive to low resolution image data in portions of the image atthe periphery of the user's field of view). In practice, providing highresolution image data within the entirety of the field of view mayconsume an excessive amount of processing and optical resources withindisplay 14, particularly given that users are only sensitive to highresolution image data near the center of the field of view. Display 14may therefore be a foveated display that displays only critical portionsof an image at high resolution to help reduce the burdens on system 10.

In general, increasing the physical size of display module 14A (e.g.,display panel 38 of FIG. 3 ) will increase the maximum resolution of theimages that can be displayed using light 22. However, space is often ata premium in compact systems such as system 10 of FIG. 1 . It wouldtherefore be desirable to be able to provide high resolution imageswhile also conserving processing and optical resources in system 10 andwithout further increasing the size of display module 14A (e.g., displaypanel 38).

In order to provide high resolution images without undesirably burdeningthe resources of system 10 and without further increasing the size ofdisplay module 14A, lens 34 may be configured to perform staticfoveation operations on light 22. Lens 34 may, for example, convertimages in the light 22′ received from display module 14A into staticallyfoveated images in light 22, which are then conveyed to the eye box(e.g., the light 22 conveyed to eye box 24 by waveguide 26 of FIG. 2 mayinclude statically foveated images). The statically foveated images mayinclude high resolution region(s) and low resolution region(s) thatcorrespond to the pixels in the images in light 22′. Lens 34 may createthe high resolution and low resolution regions in the staticallyfoveated images by using a non-uniform magnification as a function ofangle within the field of view (e.g., the magnification of lens 34 mayvary as a function of angle θ relative to optical axis 39 within itsfield of view).

FIG. 4 is a diagram showing a statically foveated image that may beproduced by lens 34 based on image light 22′ of FIG. 3 . Light 22′ mayinclude an image (e.g., as produced by pixel array 40 in display panel38 upon reflection of illumination light from light source 36). Theimage may include pixels. Lens 34 may magnify light 22′ and thus theimage in light 22′ with a magnification (optical power) that varies as afunction of angle within the field of view of lens 34 (and thus as afunction of pixel position in the image).

For example, lens 34 may magnify the image in light 22′ with amagnification that varies as a function of angle within its field ofview to produce statically foveated image 44 of FIG. 4 . As shown inFIG. 4 , statically foveated image 44 may include lower resolutionpixels 50 in regions 48 and higher resolution pixels 50 in one or moreregions 46. Region 46 may, for example, be a central region located at acenter of the image and thus at a center of the field of view of lens34. Regions 48 may, for example, be peripheral regions that run alongthe periphery of the image (e.g., around region 46) and thus along theperiphery of the field of view.

Each pixel 50 in statically foveated image 44 may correspond to arespective pixel from the image received by lens 34 in light 22′.However, lens 34 may exhibit a higher magnification at relatively highangles within the field of view (e.g., at pixel positions correspondingto regions 48) while simultaneously exhibiting a lower magnificationnear the center of the field of view (e.g., at pixel positions withinregion 46). This may cause the pixels 50 in regions 48 to exhibit arelatively large size (pitch), whereas the pixels in region 46 exhibit arelatively small size. This configures statically-foveated image 44 toexhibit a relatively high resolution (e.g., a relatively high pixeldensity) within region 46 and a relatively low resolution (e.g., arelatively low pixel density) within regions 48.

Because statically foveated image 44 has a higher resolution withincentral region 46 than within peripheral regions 48, the user (e.g., ateye box 24 of FIG. 2 ) may perceive statically foveated image 44 as ahigh resolution image (e.g., because the user's eye is sensitive to thehigh resolution within central region 46 and is insensitive to the lowerresolution within peripheral regions 48). This may allow the imagesdisplayed at eye box 24 to effectively appear as high resolution imageswithout requiring an increase in the size of display module 14A or theprocessing and optical resources of system 10 (e.g., the foveation maybe statically performed by lens 34 without imposing any increased burdenon the other components in system 10). The example of FIG. 4 is merelyillustrative. Regions 46 and 48 may have any desired shapes and/orsizes.

Curve 52 of FIG. 5 plots pixel density as a function of angle forstatically foveated image 44 within the field of view of lens 34. Thevertical axis of FIG. 5 plots pixel density in pixels-per-degree (PPD).The horizontal axis plots of FIG. 5 plots the angle θ within the fieldof view (FoV) of lens 34 (e.g., where angle θ is measured relative tothe optical axis of lens 34), which also represents pixel positionwithin the image.

As shown by curve 52, statically foveated image 44 may have a relativelyhigh (e.g., peak) pixel density D2 at the center of the field of view(e.g., at the center of the image and the optical axis of lens 34). Thismay correspond to the relatively high resolution of statically foveatedimage 44 within region 46 of FIG. 4 . Statically foveated image 44 mayhave a reduced pixel density at relatively high angles off of the centerof the field of view (e.g., off the optical axis and near the peripheryof the field of view). For example, statically foveated image 44 mayhave a minimum pixel density D1 at angles θ1 and −θ1 off of the centerof the field of view. This may correspond to the relatively lowresolution of statically foveated image 44 within regions 48 of FIG. 4 .

As examples, pixel density D2 may be 30 PPD, 25 PPD, 20 PPD, 35 PPD,between 25 and 35 PPD, between 20 and 30 PPD, between 20 and 35 PPD,greater than 30 PPD, etc. Pixel density D1 may be 18 PPD, 20 PPD, 15PPD, between 15 and 25 PPD, between 15 and 20 PPD, between 10 and 20PPD, less than 25 PPD, less than 20 PPD, or any other density less thanpixel density D2. Angle θ1 may be 26 degrees (e.g., in scenarios wherelens 34 has a 52°×52° field of view), 25 degrees, between 25 and 30degrees, between 20 and 30 degrees, etc. Curve 52 may have any desiredroll-off (shape).

FIG. 6 is a plot showing how the magnification of lens 34 may vary as afunction of angle within the field of view to produce staticallyfoveated image 44 of FIG. 4 and curve 52 of FIG. 5 . As shown in FIG. 6, curve 54 plots the magnification of lens 34 as a function of angle θwithin the field of view. As shown by curve 54, lens 34 may exhibit arelatively low (e.g., minimum) magnification M1 at the center of thefield of view (e.g., at the center of the image and the optical axis oflens 34). Magnification M1 may be zero (e.g., no magnification) ifdesired. This low magnification may allow the pixels 50 within region 46of FIG. 4 to have a relatively high pixel density and thus a relativelyhigh resolution. Lens 34 may exhibit a relatively high (e.g., peak)magnification M2 at relatively high angles off of the center of thefield of view (e.g., off the optical axis and near the periphery of thefield of view). For example, lens 34 may exhibit a maximum magnificationM2 at angles θ1 and −θ1 off of the center of the field of view. Thishigh magnification may increase the apparent size of each pixel 50within regions 48 of FIG. 4 , thereby causing the pixels 50 withinregions 48 to have a relatively low pixel density and thus a relativelylow resolution. Curve 54 may have any desired roll-off (shape).

Lens 34 may have one or more lens elements. The number, shape, andarrangement of each of the lens elements may be selected to produce themagnification associated with curve 54 of FIG. 6 (e.g., so that lens 34produces statically foveated image 44 having a pixel density such as thepixel density associated with curve 52 of FIG. 5 ). For example, lens 34may be configured to exhibit a mapping function (image height, e.g., inmillimeters) h_(img) that is a function of angle θ within the field ofview, as given by equation (1):h _(img)(θ)=f*α*sin(θ/β)  (1)where f, α, and β are constants, “sin( )” is the sine operator, “/” isthe division operator, and “*” is the multiplication operator. Constantsf, α, and β may, for example, be determined from a parametric fit. Asjust one example, constant f may be 8.6 mm, constant α may be 0.5, andconstant β may be 0.49. This is merely illustrative and, in general,constants f, α, and β may have other values, the mapping function mayhave other forms, and the lens elements may have other arrangements ifdesired.

FIG. 7 is a diagram showing one possible arrangement that may be used toform lens 34. Lens 34 of FIG. 7 may, for example, implement the mappingfunction given by equation (1) and/or the non-uniform magnificationassociated with curve 54 of FIG. 6 , and may produce statically foveatedimage 44 of FIG. 4 (e.g., as characterized by curve 52 of FIG. 5 ).

As shown in FIG. 7 , lens 34 may include one or more lens elements 60such as a first lens element 60-1, a second lens element 60-2, and athird lens element 60-3. Lens element 60-2 may be optically interposedbetween lens elements 60-1 and 60-3. Lens element 60-3 may be opticallyinterposed between lens element 60-2 and display module 14A.

In the example of FIG. 7 , display module 14A includes display panel 38(e.g., a reflective display panel such as a DMD or LCOS panel). Prism 56(e.g., prism 42 of FIG. 3 ) may be interposed between lens element 60-3and display panel 38. If desired, lens element 60-3 and/or display panel38 may be mounted to prism 56. This is merely illustrative and, ifdesired, an emissive display panel or other types of display modules maybe used. Lens 34 and display module 14A (e.g., display 14) may, forexample, be non-telecentric.

Light 22′ (e.g., light reflected off of display panel 38 and includingan image to be displayed) may pass through lens 34, which opticallyconverts light 22′ into light 22 (e.g., lens 34 converts the image inlight 22′ into statically-foveated image 44 of FIG. 4 in light 22). Lens34 may produce light 22 (e.g., the statically foveated image 44 in light22) by applying, to light 22′, a non-uniform magnification that variesas a function of angle θ relative to its optical axis (e.g., by applyingthe magnification associated with curve 54 of FIG. 6 having greatermagnification at high angles θ and the periphery of the field of viewand lower magnification at low angles θ and the center of the field ofview to light 22′).

Lens element 60-3 may have a first surface (face) 66 facing displaypanel 38 and an opposing second surface (face) 62 facing lens element60-2. Lens element 60-2 may have a first surface 64 facing lens element60-3 and an opposing second surface 68 facing lens element 60-1. Lenselement 60-1 may have a first surface 70 facing lens element 60-2 and anopposing second surface 72. Prism 74 or other optical elements may beused to direct light 22 to waveguide 26 of FIG. 2 . Prism 74 may beomitted if desired.

The number of lens elements 60, the arrangement of lens elements 60, thetypes of lens elements 60, and/or the shapes of the surfaces of lenselements 60 (e.g., surfaces 72, 70, 68, 64, 62, and 66) may be selectedto provide lens 34 with the desired magnification profile (e.g., withthe non-uniform magnification associated with curve 54 of FIG. 6 and themapping function given by equation (1)), which configures lens 34 toproduce statically foveated image 44 (FIG. 4 ) in light 22. In thearrangement of FIG. 7 , for example, lens element 60-1 is a meniscuslens having curved surfaces 72 and 70 (e.g., free form curved surfaces,radially symmetric curved surfaces such as spherically curved surfaces,radially asymmetric curved surfaces such as aspherically curvedsurfaces, etc.), lens element 60-2 is a butterfly or V-shaped lens(e.g., having a high order aspheric surface 68 and a planar orlow-curvature surface 64), and lens element 60-3 has a planar surface 66and a curved surface 62 (e.g., a free form curved surface, a radiallysymmetric curved surface such as a spherically curved surface, aradially asymmetric curved surface such as an aspherically curvedsurface, etc.). This example is merely illustrative and, in general, anydesired lens elements 60 of any desired types may be used. The surfacesof the lens elements 60 in lens 34 (e.g., surfaces 72, 70, 68, 64, 62,and 66) may have any desired shapes (e.g., free form curved shapes,radially symmetric curved shapes such as a spherical shapes, radiallyasymmetric curved shapes such as aspheric shapes, planar shapes, shapeshaving curved and planar portions, combinations of these, etc.).

If desired, an optional diffractive optical element such as diffractiveoptical element 58 may be interposed between lens 34 and display panel38 (e.g., mounted to prism 56 and lens element 60-3). Diffractiveoptical element 58 may include a diffractive grating structure havingone or more diffractive gratings (e.g., volume holograms, thin filmholograms, surface relief gratings, three-dimensional metal gratings,etc.). The diffractive gratings may be partially or completelyoverlapping (e.g., multiplexed) or may be non-overlapping. Diffractiveoptical element 58 may be formed at other locations (e.g., between lenselement 60-1 and prism 74, between any pair of lens elements 60 in lens34, or elsewhere). Diffractive optical element 58 may diffract light 22′to provide light 22′ with an optical power (e.g., an optical powercorresponding to curve 54 of FIG. 6 or other optical powers). This mayallow lens 34 to impart more optical power to light 22′ without usingadditional lens elements, which may occupy an excessive amount of spacein device 10. In another suitable arrangement, diffractive opticalelement 58 may be omitted. A doublet of lens elements or other types oflens elements may be used in place of diffractive optical element 58 toprovide light 22′ with optical power if desired.

The examples described above in which lens 34 includes lens elements 60for performing static foveation is merely illustrative. In anothersuitable arrangement, lens 34 may include one or more portions ofwaveguide 26 (FIG. 2 ). For example, waveguide 26 may include one ormore curved surfaces or other structures in the optical path of imagelight 22 that impart different optical powers on image light 22 (e.g.,different optical powers for different portions of the image to producestatically foveated image 44 of FIG. 4 ). These portions of thewaveguide may, if desired, stretch the image light in a single dimension(e.g., a horizontal dimension). This portion of the waveguide may, ifdesired, be used to perform field of view expansion (e.g., from 30degrees to 45 degrees or more in the horizontal dimension). Lens 34 mayinclude a combination of lens elements 60 and portions of waveguide 26or may include portions of waveguide 26 without including separate lenselements 60 if desired.

If care is not taken, the non-uniform magnification imparted by lens 34in producing statically foveated image 44 may undesirably distort theimage in light 22. If desired, system 10 may perform pre-distortionoperations on the images in light 22′ that compensate for subsequentdistortion by lens 34 in operating on light 22′ (e.g., distortion causedby the non-uniform magnification of lens 34). System 10 may additionallyor alternatively perform independent control of the intensity oflight-emitting elements in display module 14A to mitigate fornon-uniform intensity across the area of statically foveated image 44.

FIG. 8 is a flow diagram showing how system 10 may perform predistortionoperations on the images in light 22′ that compensate for subsequentdistortion by lens 34 in operating on light 22′. As shown in FIG. 8 ,control circuitry 16 may include an image source such as image source 76(e.g., image source circuitry) and a pre-distortion engine (e.g.,pre-distortion circuitry) such as pre-distortion engine 80. Image source76 and pre-distortion engine 80 may, for example, be implemented usinghardware (e.g., dedicated circuitry) in control circuitry 16 and/orsoftware running on control circuitry 16.

Image source 76 may produce a high resolution image such as highresolution image 78. High resolution image 78 may include pixels 50 ofimage data. Image source 76 may provide high resolution image 78 topre-distortion engine 80, as shown by arrow 79.

Pre-distortion engine 80 may apply a distortion to high resolution image78 (sometimes referred to herein as a pre-distortion) to producepre-distorted image 82. Pre-distorted image 82 may, for example, includethe same pixels 50 of image data as high resolution image 78 but wheresome or all of the pixels are pre-distorted relative to (e.g., larger orsmaller than) the corresponding pixels in high resolution image 78(e.g., pixels 50 near the center of image 82 may be smaller than thepixels 50 near the center periphery of image 78, pixels 50 near the edgeof image 82 may be larger than the pixels 50 near the edge of image 78,etc.). The pre-distortion applied by pre-distortion engine 80 may beconfigured to mitigate subsequent distortion to the image by lens 34 ingenerating statically foveated image 44 (e.g., the pre-distortion may bean inverse of any subsequent distortion applied by lens 34 on light22′). As examples, pre-distortion engine 80 may be implemented as asoftware engine (e.g., as a program containing sets of instructions forexecution by a general purpose computing element such as a CPU and/orGPU) or from a set of fixed purpose transistors, logic gates, etc.

Display panel 84 in display module 14A may display (project)pre-distorted image 82 as projected pre-distorted image 85 in light 22′.Display panel 84 may be a reflective display panel (e.g., display panel38 of FIGS. 3 and 7 ), an emissive display panel, or any other desireddisplay panel or light source.

Lens 34 may magnify light 22′ (e.g., using a non-uniform magnificationsuch as the magnification associated with curve 54 of FIG. 6 ) toproduce statically foveated image 44 in light 22. Any optical distortionproduced by lens 34 on light 22′ may reverse the predistortion inprojected pre-distorted image 85. This may cause statically foveatedimage 44 to be non-distorted while still exhibiting a high resolutionwithin region 46 (FIG. 4 ) and a low resolution within regions 48 (FIG.4 ). Statically foveated image 44 (light 22) may be provided towaveguide 26, as shown by arrow 89. Waveguide 26 may provide light 22and thus statically foveated image 44 to the eye box (e.g., eye box 24of FIG. 2 ).

If desired, the intensity of the light-emitting elements in displaymodule 14A may be independently controlled to compensate for inherentoff-axis roll off in intensity and/or distortion from lens 34. FIG. 9 isa diagram showing how the intensity of the light-emitting elements(e.g., uLEDs, lasers, LEDs, or other light-emitting elements inscenarios where display module 14A includes an emissive display or lightemitting elements 35 of FIG. 3 ) may be independently controlled tomitigate these effects.

As shown in FIG. 9 , the horizontal axis illustrates the lateralposition of the light emitting elements in display module 14A (e.g.,horizontal or vertical pixel position across an array of M-by-N orN-by-N light-emitting elements). Curve 88 of FIG. 9 illustrates theintensity of illumination produced by the light-emitting elements (e.g.,as measured on the side of lens 34 opposite to display module 14A).Curve 90 illustrates the maximum intensity producible by thelight-emitting elements. As shown by curve 88, the illumination mayexhibit a roll off from a peak intensity at central axis C to a minimumintensity at positions off of central axis C (e.g., for pixels at theperiphery of the array of light-emitting elements). This variation inintensity may, for example, be produced by inherent off-axis roll off inintensity associated with display module 14A and/or off-axis distortionproduced by lens 34.

In order to mitigate this variation, light-emitting elements located offof central axis C (e.g., at the periphery of the array) may beindependently controlled to emit light with an increased intensity, asshown by arrows 96. This boost in peripheral pixel intensity may provideillumination with a uniform intensity for each light-emitting elementposition by the time the light has passed through lens 34. In anothersuitable arrangement, the light-emitting elements located at centralaxis C may be independently controlled to emit light with decreasedintensity (e.g., with an intensity that matches that of thelowest-intensity pixels), as shown by arrow 94. This reduction incentral pixel intensity may provide illumination with a uniformintensity for each pixel position by the time the light has passedthrough lens 34. These adjustments in intensity may be provided byadjusting the current provided to each light-emitting element, byadjusting the pulse width modulation used to control each light-emittingelement, etc. By independently controlling the intensity of eachlight-emitting element as a function of position, light of uniformintensity may be provided despite distortions introduced by opticalsystem 14B. The example of FIG. 9 is merely illustrative. Curves 88, 90,and 92 may have other shapes.

FIG. 10 is a flow chart of illustrative steps that may be performed bysystem 10 in performing static foveation operations. At step 100, imagesource 76 may provide high resolution image 78 to pre-distortion engine80.

At step 102, pre-distortion engine 80 may pre-distort high resolutionimage 78 to produce pre-distorted image 82. Control circuitry 16 mayprovide pre-distorted image 82 to display module 14A (e.g., displaypanel 84).

At optional step 104, control circuitry 16 may independently control theintensity of each light-emitting element in display panel 14A tomitigate for any intensity variations across the field of view (e.g., asdescribed above in connection with FIG. 9 ). Step 104 may be omitted ifdesired.

At step 106, display panel 84 may display pre-distorted image 85 inlight 22′.

At step 108, lens 34 may receive displayed pre-distorted image 85 inlight 22′. Lens 34 may magnify light 22′ (pre-distorted image 85) usingdifferent magnifications at different pixel positions (e.g., using themagnification associated with curve 54 of FIG. 6 ) to produce staticallyfoveated image 44 in light 22.

At step 110, waveguide 26 (FIG. 2 ) may receive the light 22 includingstatically foveated image 44. Waveguide 26 may direct light 22 and thusstatically foveated image 44 to eye box 24. In this way, the user mayview statically foveated image 44 and may perceive the image as a highresolution image, despite the lower resolution of pixels near theperiphery of the image. This may serve to maximize the effectiveresolution of system 10 without increasing the processing or opticalresources required to display light 22 and without increasing the sizeof display module 14A.

The systems and methods described herein for producing staticallyfoveated image 44 (FIG. 4 ) is merely illustrative. Additionally oralternatively, these systems and methods may be used to expand the fieldof view the image light provided at eye box 24.

In accordance with an embodiment, a display system is provided thatincludes a display panel having a pixel array, a light source thatilluminates the pixel array to produce image light that includes animage, the image has a central region of pixels and a peripheral regionof pixels surrounding the central region of pixels, a waveguide, and alens configured to receive the image light, the lens is furtherconfigured to direct the image light towards the waveguide whileapplying a first magnification to the pixels in the peripheral region ofthe image and a second magnification to the pixels in the central regionof the image, the first magnification is greater than the secondmagnification, and the waveguide is configured to direct the image lighttowards an eye box.

In accordance with another embodiment, the display panel includes adisplay panel selected from the group consisting of adigital-micromirror device (DMD) panel and a liquid crystal on silicon(LCOS) panel.

In accordance with another embodiment, the display panel includes anemissive display panel.

In accordance with another embodiment, the waveguide includes volumeholograms configured to diffract the image light towards the eye box.

In accordance with another embodiment, the lens is characterized by amapping function, the mapping function being a function of the sine ofan angle within a field of view of the lens divided by a constant value,and the angle being measured with respect to an optical axis of thelens.

In accordance with another embodiment, the lens includes first, second,and third lens elements, the first lens elements being interposedbetween the second lens element and the display panel, and the secondlens element being interposed between the first and third lens elements.

In accordance with another embodiment, the first lens is a meniscuslens.

In accordance with another embodiment, the second lens is a butterflylens.

In accordance with another embodiment, the first lens element has a freeform curved surface.

In accordance with another embodiment, the display system includes adiffractive optical element interposed between the lens and the displaypanel, the diffractive optical element being configured to provide anoptical power to the image light.

In accordance with another embodiment, the display system includes apre-distortion engine configured to apply a pre-distortion to the imagein the image light produced by the display panel, the lens applies adistortion to the image light, and the pre-distortion is an inverse ofthe distortion applied by the lens.

In accordance with another embodiment, the light source includes firstand second light-emitting elements, the display system includes controlcircuitry, the control circuitry is configured to control the firstlight-emitting element to illuminate the pixel array with a firstintensity of light, and the control circuitry is configured to controlthe second light-emitting element to illuminate the pixel array with asecond intensity of light that is different from the first intensity.

In accordance with an embodiment, an electronic device is provided thatincludes an image source configured to produce an image, apre-distortion engine configured to generate a pre-distorted image byapplying a pre-distortion to the image, a display module configured todisplay light that includes the pre-distorted image, a lens having afield of view, the lens is configured to receive the light that includesthe pre-distorted image from the display module, the lens is configuredto produce a foveated image based on the pre-distorted image by applyinga non-uniform magnification to the light, and the non-uniformmagnification varies as a function of angle within the field of view,and a waveguide configured to direct the foveated image towards an eyebox.

In accordance with another embodiment, the predistortion compensates fora distortion associated with the non-uniform magnification applied tothe light by the lens.

In accordance with another embodiment, the field of view of the lens hasa central region and a peripheral region surrounding the central regionand the non-uniform magnification includes a first amount ofmagnification within the central region and a second amount ofmagnification within the peripheral region, the second amount ofmagnification being greater than the first amount of magnification.

In accordance with another embodiment, the foveated image has a firstresolution within the central region and a second resolution within theperipheral region, the second resolution being less than the firstresolution.

In accordance with another embodiment, the electronic device includescontrol circuitry, the control circuitry is configured to independentlycontrol intensities of light-emitting elements within the display moduleto mitigate for non-uniform intensity in the light.

In accordance with another embodiment, the lens includes a portion ofthe waveguide.

In accordance with an embodiment, an electronic device is provided thatincludes a head-mounted support structure, a display module supported bythe head-mounted support structure, the display module is configured toproduce light that includes an image, a waveguide supported by thehead-mounted support structure, and a lens that is configured direct thelight towards the waveguide and that has an optical axis, the lens isconfigured to produce a foveated image in the light by applying, to theimage in the light, a first magnification at a first angle with respectto the optical axis and a second magnification at a second angle withrespect to the optical axis, the first angle is smaller than the secondangle, the first magnification is less than the second magnification,and the waveguide is configured to direct the foveated image in thelight towards an eye box.

In accordance with another embodiment, the display module includes aspatial light modulator and a light source that is configured toilluminate the spatial light modulator to produce the light thatincludes the image.

The foregoing is merely illustrative and various modifications can bemade to the described embodiments. The foregoing embodiments may beimplemented individually or in any combination.

What is claimed is:
 1. A display system comprising: a display panelhaving a pixel array; a light source that illuminates the pixel array toproduce image light that includes an image, wherein the image has acentral region of pixels and a peripheral region of pixels surroundingthe central region of pixels; a waveguide; and a lens configured toreceive the image light, wherein the lens is further configured todirect the image light towards the waveguide while applying a firstmagnification to the pixels in the peripheral region of the image and asecond magnification to the pixels in the central region of the image,wherein the first magnification is greater than the secondmagnification, and wherein the waveguide is configured to propagate theimage light as a foveated image via total internal reflection.
 2. Thedisplay system defined in claim 1, wherein the display panel comprises adisplay panel selected from the group consisting of: adigital-micromirror device (DMD) panel and a liquid crystal on silicon(LCOS) panel.
 3. The display system defined in claim 1, wherein thedisplay panel comprises an emissive display panel.
 4. The display systemdefined in claim 1, wherein the waveguide comprises volume hologramsconfigured to diffract the image light.
 5. The display system defined inclaim 1, wherein the lens is characterized by a mapping function, themapping function being a function of the sine of an angle within a fieldof view of the lens divided by a constant value, and the angle beingmeasured with respect to an optical axis of the lens.
 6. The displaysystem defined in claim 1, wherein the lens comprises first, second, andthird lens elements, the first lens element being interposed between thesecond lens element and the display panel, and the second lens elementbeing interposed between the first and third lens elements.
 7. Thedisplay system defined in claim 6, wherein the first lens element is ameniscus lens.
 8. The display system defined in claim 7, wherein thesecond lens element is a butterfly lens.
 9. The display system definedin claim 6, wherein the first lens element has a free form curvedsurface.
 10. The display system defined in claim 1, further comprising:a diffractive optical element interposed between the lens and thedisplay panel, the diffractive optical element being configured toprovide an optical power to the image light.
 11. The display systemdefined in claim 1, further comprising: a pre-distortion engineconfigured to apply a pre-distortion to the image in the image lightproduced by the display panel, wherein the lens applies a distortion tothe image light, and wherein the pre-distortion is an inverse of thedistortion applied by the lens.
 12. The display system defined in claim1, wherein the light source comprises first and second light-emittingelements, the display system further comprising: control circuitry,wherein the control circuitry is configured to control the firstlight-emitting element to illuminate the pixel array with a firstintensity of light, and wherein the control circuitry is configured tocontrol the second light-emitting element to illuminate the pixel arraywith a second intensity of light that is different from the firstintensity.
 13. An electronic device comprising: an image sourceconfigured to produce an image; a pre-distortion engine configured togenerate a pre-distorted image by applying a pre-distortion to theimage; a display module configured to display light that includes thepre-distorted image; a lens having a field of view, wherein the lens isconfigured to receive the light that includes the pre-distorted imagefrom the display module, wherein the lens is configured to produce afoveated image based on the pre-distorted image by applying anon-uniform magnification to the light, and wherein the non-uniformmagnification varies as a function of angle within the field of view;and a waveguide configured to propagate the foveated image via totalinternal reflection.
 14. The electronic device defined in claim 13,wherein the predistortion compensates for a distortion associated withthe non-uniform magnification applied to the light by the lens.
 15. Theelectronic device defined in claim 14, wherein the field of view of thelens has a central region and a peripheral region surrounding thecentral region and wherein the non-uniform magnification comprises afirst amount of magnification within the central region and a secondamount of magnification within the peripheral region, the second amountof magnification being greater than the first amount of magnification.16. The electronic device defined in claim 15, wherein the foveatedimage has a first resolution within the central region and a secondresolution within the peripheral region, the second resolution beingless than the first resolution.
 17. The electronic device defined inclaim 16, further comprising: control circuitry, wherein the controlcircuitry is configured to independently control intensities oflight-emitting elements within the display module to mitigate fornon-uniform intensity in the light.
 18. The electronic device defined inclaim 13, wherein the lens comprises a portion of the waveguide.
 19. Anelectronic device comprising: a head-mounted support structure; adisplay module supported by the head-mounted support structure, whereinthe display module is configured to produce light that includes animage; a waveguide supported by the head-mounted support structure; anda lens that is configured to direct the light towards the waveguide andthat has an optical axis, wherein the lens is configured to produce afoveated image in the light by applying, to the image in the light, afirst magnification at a first angle with respect to the optical axisand a second magnification at a second angle with respect to the opticalaxis, wherein the first angle is smaller than the second angle, whereinthe first magnification is less than the second magnification, andwherein the waveguide is configured to propagate the foveated image viatotal internal reflection.
 20. The electronic device defined in claim19, wherein the display module comprises a spatial light modulator and alight source that is configured to illuminate the spatial lightmodulator to produce the light that includes the image.